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1. Introduction 


Software testing may be a crucial a part of software package development in delivering a high-quality software package 
that's free from bugs and defects and also the method of automating software package testing is vital to its success. Testing 
is vital as a result of software package reliability is outlined exploitation testing and more or less fifty per cent of the software 
package development take into account software package comes is spent on testing. Software package testing is labour in- 
tensive and expensive; so, there's a necessity to cut back human testing. Software package testing is important as a result of 
errors is typically introduced into software package unknowingly because it is designed and made. Software package has be- 
come even more complicated nowadays, which suggests there are additional lines of code, and additional thorough testing 
that has to be done.[1] 

His purpose of testing is quality assurance, verification, and validation or dependableness estimation. It’s a trade-off 
between budget, time and quality. Computer code Quality is that the central concern of computer code engineering. Testing 


is that the single most widely used approach to making sure computer code quality. [2] 
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Testing may be additional classified into purposeful and Performance testing. Performance testing analyses if the per- 
formance parameters like work time, dependableness, load capability etc. are up to the shopper expectations. Functional 
testing aims to visualize, if the functionalities provided are correct as per business necessities and are operating as intended. 
It must be critically analysed if the investment on testing could give an acceptable come. Though’ it's continuously recom- 
mended to travel for testing, timelines and budget typically makes the project team skip an equivalent. The main objective of 
the research paper is that it describes concisely what the research is trying to achieve. They summarize the accomplishments 
a researcher wishes to achieve through the project and provides direction to the study. 


2. SOFTWARE TESTING STRATEGIES 


Software testing is that the execution of code mistreatment mixtures of input and state chosen to reveal bugs. In this section 
we tend to introduce to manual testing and automatic testing. In orders to scale back the price of manual computer code 


testing researchers are operating towards increasing the automation of software testing. 


2.1. Manual Testing 


Manual Testing may be a kind of code take a look acting during which test cases area unit dead manually by a tester while 
not victimisation any machine-driven tools. the aim of Manual Testing is to spot the bugs, issues, and defects within the code 
application. Manual code testing is that the most primitive technique of all testing varieties and it helps to seek out essential 
bugs within the code application. 

Any new application should be manually tested before its testing will be machine-driven. Manual code Testing needs 
additional effort however is critical to ascertain automation practicable. Manual testing ideas doesn't need information of 
any testing tool. One in all the code Testing elementary is "100% Automation isn't possible". This makes Manual Testing im- 


perative. 


2.2. Automation testing 


Automation testing/test Automation could be a code testing technique that performs exploitation special machine-driven 
testing code tools to execute a test suit suite. On the contrary, Manual Testing is performed by somebody's sitting before of a 
pc fastidiously capital punishment the check steps. The automation testing code may also enter test knowledge into the Sys- 
tem underneath check, compare expected and actual results and generate careful check reports. Check Automation demands 
extended investments of cash and resources. Successive development cycles would require execution of same check suite 
repeatedly. Employing a check automation tool, it's potential to record this check suite and re-play it as needed. Once the 
check suite is machine-driven, no human intervention is needed. This improved ROI of check Automation. The goal of Auto- 


mation is to scale back the amount of check cases to be run manually and to not eliminate Manual testing altogether. 


Table 1. Difference between Manual testing and Automated Testing 





Manual Testing 


Automation Testing 





Huge investment in human resources: As take a look at cases ought to 
be dead manually therefore additional testers are needed in manual 
testing. 


Less investment in human resources: take a look at cases are execut- 
ed by mistreatment automation tool thus less tester is needed in 
automation testing 





Time intense and tedious because the take a look at cases are dead 
manually, it is very slow and tedious. 


Fast: machine-driven take cases could be dead tons quicker than 
manual counterparts. 





For executing test cases for small number of times. 


To run the test multiple number of times. 





It permits the tester to perform additional ad-hoc (random testing) 


Helps acting "compatibility testing" - testing the code on totally dif- 
ferent configurations 





Manual testing is a smaller amount reliable as tests might not be 
performed with exactitude anytime because of human errors. 


Automation tests execute exactly same operation when they're run. 








The longer tester spends testing a module the grater the percentages 





Capable of finding only expected defects. 
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to search out real user bugs 














Short term costs are reduced. Long Term Cost reduction. 
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Figure 1. The Division of the tools 


Table 2. The Division of the tools 











Tools Description 
Test Design Tools Tools that assist you decide what tests got to be dead. take a look at information and action generators. 
GUI GUI take a look at Drivers Tools that alter execution of tests for merchandise with graphical user interfaces. 


Client/server take a look at automation tools, as well as load testers, additionally go here. 





Load and Performance Tools | Tools that specialise in putt an important load on systems (especially client-server systems). These tools are 
usually additionally graphical user interface take a look at drivers. 

















Test Management Tools Tools that alter execution of tests for merchandise while not graphical user interfaces. Also, tools that assist you 
work with massive take a look at suites. 
Test Implementation Tools Miscellaneous tools that assist you implement tests. as an example, tools that automatically generate stub rou- 
tines go here, as do tools that arrange to create failures more obvious (assertion generators, etc.) 
Test analysis Tools Tools that assist you value the standard of your tests. Code coverage tools go here. 
Static Analysis Tools Tools that analyse programs while not running them. Metrics tools fall during this class 











3. Testing Tools 


3.1. 


1. 


Quick Test Professional (QTP) 

Vendor-HP code Division: - It is illustrious to be the industries best answer for practical testing and regression check 
automation. It uses keyword driven approach proving the check automation engineer full access to underlying check 
associate degree object properties via an integrated scripting and debugging interface. 


2. Open XML Report Format for check Results: Stores test ends up in associate degree open XML format, enabling you 


to simply customize the reports in keeping with your own necessities, and to integrate the check result data with dif- 


ferent applications. Check results will currently be exported to mark-up language. 


3. New IDE Environment: Offers extremely customizable check development surroundings. 


4. New Debugger: allows checkers to pin-point test errors when building and maintaining check cases. Keyword Man- 


agement: Manage keywords, including turning on/off specific ways from the Keyword read. 


5. Environment Support: Quick Test skilled supports functional testing of all common environments, including Windows, 


Web, .Net, Visual Basic, ActiveX, Java, SAP, Siebel, Oracle, PeopleSoft, and terminal emulators. New Environment 
Support: Supports net services, .NET 2.0, Firefox 1.5, Netscape 8, Macromedia Flex a pair of, Win XP 64-bit, Internet 
somebody seven, and also the latest ERP/CRM applications. 
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6. Scripting Language: the selection of VBScript isn't terribly good, as a result of VBScript has some important limitations: 
like, it doesn't support modules/libraries and can't be thought-about as "truly" object-oriented language because it 
doesn't support inheritance of categories.[1] 


3.2. RANOREX 

a) Vendor- Ranorex GmbH: Ranorex is that the quickest growing automation tool that has imbibed the options of the 
simplest automation tools out there in market and providing a foothold over them. 

b) Object Repository: it's a extremely outlined object repository which properly stores and displays the objects within 
the hierarchy of their presence on the screen. 

c) User Codes: It permits the user to make any reasonably state of affairs using committal to writing that adds to any 
feature already gift within the tool. 

d) Code Compiler: Brings out any code connected issue within the take a look at script whereas assembling the code 
for execution. 

e) Enhanced Keyword View: Elaborate keyword read with all the functionalities and add-ons. 

f) Graphical Report: The take a look at report generated is complimented with graphs for quicker and higher compari- 
son of defects in each run. 

g) Step read Scripting: The interface provided could be a non-code-based interface that makes it simple to trace miss- 
ing steps. 

h) Environment Support: Having a extremely developed image based processing; it will support any graphic primarily 
based environment. 

i) Scripting Language: It supports extremely evolved languages like C#, vb.net that provides it a footing over different 
testing tools. Also, it provides the tester ability to code in his comfort language.[1] 


3.3. LambdaTest 


Cloud LambdatTest is cloud-based cross browser testing platform that helps in each manual and automatic browser compati- 
bility testing. Users will perform live interactive testing of their web site or internet app on a mixture of 2000+ totally differ- 
ent browser and software package right from their own browser. Additionally, the platform permits the users to run chemical 
element automation tests on a climbable, secure, and reliable cloud-based chemical element grid and perform live interac- 
tive cross-browser testing of their public or regionally hosted internet sites and web app on the cloud. LambdaTest addition- 
ally offers the feature of taking machine-driven page screenshots across all 2000+ environments to quickly take a look at the 
layout, sign up one click however your web site can look out on thirty six totally different devices, and compare style and 
mark-up language pictures. Additionally, the LambdaTest platform additionally has single-click integration with standard 
project management and enterprise tools like Jira, Asana, Trello, Github, Gitlab, BitBucket, Slack, and Visual Studio Team Ser- 
vices. “Built amorously for Testers!” [12] 

Cross-browser testing is one in all the essential elements to run an online application systematically on multiple brows- 
ers. it's a testing technique that enables you to see the net application practicality and alternative dependencies across nu- 
merous browsers over the net. Now, you need to be thinking is it extremely necessary to perform cross-browser testing for 
an online application. Well, as you may understand, that there square measure a range of internet browsers obtainable 
within the market, and end-users use completely different browsers to go to an online application in line with their conven- 


ience. Hence, it's necessary that the net application or web site works on all the browsers effectively[3]. A web app typically 
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has multiple client-side elements like Flex, JavaScript, Flash, Applets, and lots of additional. All of those elements will behave 
otherwise across completely different internet browsers in line with their rendering engines. Cross-browser testing ensures 
that your internet app works unfailingly across all browsers while not compromising with its quality. It permits developers 


and testers to spot whether or not the rendition or look of web content in several browsers and devices is same or not. 
3.4. TestComplete 


TestComplete is an automatic UI Testing Tool developed by SmartBear software package. TestComplete has the flexibility to 
make machine-driven useful tests for desktop, web, and mobile applications. It conjointly has native support for behavioural 
Driven Development (BDD). This tool is alleged to be terribly straightforward to use with computer science (Al). it's quick and 
straightforward to make, maintain, and execute useful take a look ats which is able to increase test coverage and make sure 
you ship high-quality, battle-tested software package. It has unmatched hybrid beholding engine with computer science. Its 
beholding combines property-based and Al-powered visual recognition to seek out dynamic Ul components quickly, easily, 
and accurately. It is conjointly known as a Script or Scriptless tool. the rationale behind this can be that if you wish to script, 
you'll presently do thus in seven completely different languages (JavaScript, Python, VBScript, Jscript, DelphiScript, C#, and 
C+). If you are doing not wish to script, then you'll use its record and play feature 


3.5. Katalon Studio 


Katalon Studio is one amongst the rising tools for automatic testing. in conjunction with atomic number 34, Ranorex, Test- 
Complete, and Apache JMeter, it’s price discussing as additional and additional testing engineers pick Katalon. The platform 
facilitates running automatic tests for net interfaces, APIs, and mobile apps (both iOS and Android), provides check recording, 
and analysis reports. Katalon Studio could be a free-license tool discharged in January 2015 with a Selenium-based engine. 
Mostly, Katalon is intended to form and utilise automatic check scripts for UI while not committal to writing. Katalon Studio 
permits running automatic tests of Ul parts, together with pop-ups, iFrames, and wait-time. The tool is launched on Mi- 
crosoft Windows, macOS, and Linux. 

Katalon’s main advantage is that it’s straightforward to deploy and features a wider set of integrations compared to Se, 
the market leader. Katalon has twin scripting interfaces for users with totally different programming skills. this suggests that 
testers with restricted technical data will use an easier program that doesn’t need writing code. The mode for enhanced us- 
ers has access to scripting with syntax lightness, code suggestion, and debugging. Katalon Studio has pre-defined whole 
structure: variety of templates for check cases, check suites, check objects, and reports. 

Katalon Studio supports native and remote testing, still as parallel and consecutive executions. It runs on Groovy (Java) 
scripting language. This answer features a fast setup and variety of pre-installed templates that enable continuation some 
testing patterns. Katalon Studio may be a cross-browser tool that supports internet, mobile, and API testing. These solutions 
escort analytics and recording modules 


3.6. SELENIUM 


Selenium could be a free (open-source) machine-controlled testing framework want to validate internet applications across 
totally different browsers and platforms. you'll be able to use multiple programming languages like Java, C#, Python etc to 
make chemical element take a look at Scripts. Testing done mistreatment the chemical element tool is sometimes said as 
chemical element Testing. Selenium software package isn't simply one tool however a collection of software package, every 
bit job to totally different testing wants of a company. Here is that the list of tools Selenium Integrated Development sur- 
roundings (IDE) Selenium remote (RC) WebDriver Selenium Grid.[9] 


3.7. API 
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Application Programming Interface (API). Ports take a look at scripts you write in like Ruby, Java, Python, or C# to Selenium’s 


own scripting language (Selenese), through bindings. 


e Library: Houses the API and language-specific bindings. though many third-party bindings exist to support totally 


different programming languages, the core client-side bindings supported by the most project are: element Java (as 


element jar files), element Ruby, element dotnet (or element C#, out there as .dll files), element Python, and ele- 


ment JavaScript (Node). 


e Driver: Executable module that parades a browser instance and runs the take a look at script. Browser-specific—for 


instance, Google develops and maintains Chrome driver for element to support automation on Chromium/Chrome. 


Framework Support libraries for integration with natural or artificial language take a look at frameworks, like ele- 


ment with Cucumber or element with TestNG. 


Table 3. A Comparison Software Testing Tools 


























Features TestComplete SELENIUM Quick Test Profes- | Katalon Studio RANOREX LambdaTest 
sional 
Test devel- | Windows Cross platform Windows Cross platform Cross platform Cross __ plat- 
opment form 
platform 
Application Mobile, Web, | WebApplications | Mobile, Web, Win- | Mobile, Web, | Mobile, Web, | Web Appli- 
under test Windows  desk- dows desktop, apps, | Windows desktop, | Windows desktop, | cations 
top, apps, API/Web services apps, API/Web | apps, API/Web 
API/Web services services services 
Scripting Java, C#, Python | Java, C#, Python | VBScript Java VB.NET and C# Java, C#, 
languages etc. etc. Python etc. 
Programming | Not required Advanced skills | Not required. Not required Not required Not re- 
skills needed to inte- | Recommended for quired 
grate various | advanced test scripts 
tools 
Script crea- | Fast Slow Fast Fast Slow Fast 
tion time 























4. RESEARCH TOOLS 


Research tools can be defined as vehicles that broadly facilitate research and related activities. Scientific tools enable re- 


searchers to collect, organize, analyse, visualize and publicized the research outputs. 


4.1. PERFORMANCE TESTING TOOLS 


The top performance testing is been search and as listed below some of them. The diagram representation of performance 


testing is below. 


ais Performance Testing oe 
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Figure 2. Performance Testing 
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In terms of performance, we wish to achieve high speed, scalability and stability of a system. There are several types of 


performance tests in which each simulates different possible scenario. The below figure demonstrates that how the some of 











them works: 
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Figure 3. Analysis of Performance Test Techniques 


Importance of Performance Testing in Web Applications: 
Scalability: Not only speed an important goal for performance, but scalability tests are extremely important if you 


want more users to interact with a system. 
Stability: Obviously, we want your application to work at all times. There will be some times when it is under more 


stress than others. 


4.1.1.WebLOAD 


WebLOAD is the tool of choice for enterprises with heavy load and complex testing requirements. It allows you to perform 


load and stress testing on any internet application by generating load from the cloud and on premises machine. 
WebLOAD’s strengths are its flexibility and ease of use is — enabling you to quickly define the tests you need with fea- 


tures like DOM — based recording/playback, automatic correlation, and JavaScript scrioting language. WebLOAD’s supports 


the hundreds of technologies — web protocol to enterprise applications and has buit in integrations with Jenkins, Selenium 


and many other tools to enable continupus load testing for DevOps. 


4.1.2.LoadNinja 
LoadNinja by SmartBear allows you to quickly create scriptless sophisticated load tests, reduce testing time by 50%, replace 


load emulators with real browsers, and get actionable, browser-based metrics, all at ninja speed.The automated perfor- 


mance software testing tools which are currently available are [9]. 
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4.1.3.HP Load Runner 


HP Load Runner is the product of HP. It is an industrial standard based performance automated tool for load testing of appli- 
cations which is used for studying system performance and behavior. It works by using virtual users. It simulates thousands of 
real-time users to put the application through user loads and thoroughly analyses the results to identify the particular be- 


havior. 
4.1.4.Apache J Meter 


Apache J Meter is a product of Apache Software Foundation designed to load test functional behavior and measure perfor- 
mance. It is an open source java application. Apache JMeter was basically designed for testing web applications but now it 
can be used for performance test both on static and dynamic resources. Distributed testing feature is one of the strong 
points of Apache J Meter. 


Aggregete Report 
wae 
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Figure 4. The apache J Meter works and then it results in a tabular form that what total data is there in particular software. 


4.1.5.Selenium 


Selenium is used to test web application. It is consists of four tools i.e. Selenium IDE, Selenium RC, Selenium Web Drive and 
Selenium Grid. Selenium is an open license and supports different web browsers. 


4.1.6.NeoLoad 


NeoLoad is used to measure the performance of the web application. NeoLoad provides pragmatic solution facilitating design 
and development of the optimized websites. Neotys is a French company which owned, maintained and developed NeoLoad. 
It provides monitoring of the user response times and infrastructures statistics. It can run on most of the operating systems. 
A non-specialist can easily create a test by using the NeoLoad automation testing tool. 


4.1.7.WebLoad Professional 


WebLoad Professional is developed by Rad Views for Performance testing of internet applications. WebLoad Professional 
includes built-in support for Ajax technologies, JSON data types and different types of SOAP and XML web services. It sup- 
ports Windows operating systems as well as Linux machines. The proxy recorder is used for recording business processes. 
The WebLoad result test script is written in JavaScript. It is an open source project. 
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4.1.8.LoadUI 


LoadUI is an open source performance testing tool used for load testing. LoadUI is used to check the performance of web 
application. LoadUI can also work withSoapUI really efficiently. LoadUI is the most flexible and interactive performance test- 
ing tool. 


4.1.9.WAPT 


Web Application Performance Tool or WAPT, is used to test web application and the interface of web. WAPT is used for per- 
formance, Load and Stress Testing of web application. WAPT specify to the tester that how many virtual users are using the 


testing environment i.e. either increasing, constant or interrupted users load. 
4.1.10. Loadster 


Loadster is a Load Testing Tool which is used for test solutions for websites, web application and web services. Loadster is 
built for real web applications and handling cookies, user sessions, custom header, dynamic form data. Loadstar creates a 
testing environment where single user state and Loadster gathers statistics for each virtual user individually. 


4.1.11. LoadImpact 


LoadImpact is a performance testing tool where it simulate traffic exactly how it would happen in real life in a testing envi- 
roment. Loadimpact generate scripts automatically and there is no need of programming. Loadimpact also measure the us- 
age of CPU, Memory Usage, Disk I/O and Network I/O. 


5. Identifying Usability requirements of Software Testing Tools 


According to Andreas Hollinger, usability is defined as the comfort of use and suitability of a system for a particular class of 
users carrying out precise work in a precise environment. Author specified criteria level for ease of use, ease of learning, sim- 
plicity, effectiveness, information, and the user interface as usability requirements for the evolution of selected software 


testing tools. 
5.1. Selection of Participants 


For conducting the usability studies, the author selected a total of 20 participants. These participants are students of Blekinge 
Institute of Technology who are currently studying in the field of Computer Science and Software Engineering. Author divides 
the participants into two groups. Group 1 will conduct the usability test using LoadRunner while group 2 will conduct the us- 
ability test using JMeter. The participants were selected randomly. Initially 28 Students were contacted for the survey but 
only 23 students agreed. Later on, 2 students informed the author that they are busy and would not be able to attend the 
survey and 1 participant didn“t show up for the survey. 28 In these 20 participants, 4 participants have work experience in 
software testing and are familiar with automated testing tools. 6 students have already the experience of conducting the 
usability survey and they are equally divided into the groups. Remaining 10 participants have no experience in usability sur- 
vey and are divided equally into two groups. 


5.2. Scenarios based Tasks 


Tasks selected for the usability test are a scenario based and is written in the English language. These tasks cover all the func- 
tionalities of the selected parameters of the selected tools. It is not easy to test all the functionalities of the selected tools so 


author designed the scenarios in such way that the objective of this research is achieved. 


5.3. Test Location 
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The usability test is conducted in Blekinge Institute of technology group rooms. All the tasks were provided on paper and par- 


ticipants were left alone so that they can conduct the test without any interruption. 


5.4. Usability Test Equipment and Material 


To have a control experiment, author provides computer to every participant. The Same computer was provided to each par- 


ticipant. After the completion of the test, the author made sure that the software testing tool is completely uninstalled. All 


the tasks and questionnaires were written on paper and were provided to each participant individually. 


Table 4. Comparison Between LoadRunner and JMeter Tool 











Parameters LoadRunner JMeter 
Procurement Cost Licensed tool with high maintenance cost Open-Source free tool 
Installation Time consuming High disk usage Quick Low disk usage 





Load Generation 


Restricts number of users although licensed tool 


No restriction on number of users. Allows as many 
users as hardware supports 








Scripts Hides scripts in levels in order to make script look | Shows entire script describing the HTTP flow of the 
better scenario 
Result Excellent analysis, dynamic graph generation, serv- | Limited graph generation with no server-side met- 


er-side metrics 


rics 





Transaction Response Time 


More as response time includes image download time 


Less as it excludes image download time 





Analysis and Monitoring 


Features provided are market leading 


Not very strong in this category 





Record and Replay Feature 


Provides unmatched record and replay feature com- 
pared to any open source 


Record and replay feature not very strong 





Preferred Use 


When working with different types of IT applications, 
dealing with different protocols 


When working extensively on web protocols 








Resolutions 





Faster 





Slow as testers have to find ways to resolve issues 








Graph Representation 








@ loadrunner 


mj meter 








ISSN (Online) : 2582-7006 


Journal of Informatics Electrical and Electronics 


10 


Engineering (JIEEE) 


A2Z Journals = 


@ 


Va 


N. Srivastava et al. 


e —_ Line Representation 





=== LoadRunner 
=———J Meter 


2 == Column1 


Category 1 Category 2 Category3 Category4 Category5 category 6 





6. WEB APPLICATION PERFORMANCE TESTING 


This chapter discusses the test preparation, execution and reporting of web application performance testing implementation. 
The main focus of this experiment is load testing. The below figure shows the web application architecture 


bee! 


Figure 5. Web Application Testing 





7. Conclusion 


In this research paper one point is indubitable that automation testing is much more suitable than manual testingTesting is 
one among the helpful and necessary side once it comes to testing the applications. Open supply technologies are becoming 
common day by day due to the community support to those open supply software system. Here during this paper when 
evaluating the 3 preferred Open supply Softwares viz. checkComplete, SELENIUM, Quick Test Professional, Katalon Studio, 
RANOREX and LambdatTest. 

Software performance testing has become an integral part of projects. HP LoadRunner and Apache JMeter are good 
performance testing tools available in the market but which tool is ultimately best depends on a number of factors like your 
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budget, type of system, testers, no of users etc. The low cost advantage may not be very helpful if we are dealing with legacy 
systems where lots of scripts are developed, compared to the efforts required for rescripting. On the other hand if we are 
working on different types of applications and using different protocols then going with LoadRunner will be preferred as it 
supports many protocols. Based on this research we recommend to go ahead with HP LoadRunner as it is very stable and 
robust. LoadRunner provides faster resolution of scripting issues. Recent study says that, efficiency of software professionals 
working on licensed tool is more than those working on open source tools. LoadRunner provides excellent monitoring and 
analysis features which are leading in the market. Its only disadvantage is high procurement and maintenance cost. But if you 


have budget crisis then you can stick to JMeter. 
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